Shape extraction system and 3-D (three dimension) information acquisition system using the same

ABSTRACT

A background lighting module illuminates an object from behind by visible light with respect to a photographing module to identify an area including the boundary between the object and the background portion in an image to be photographed by the photographing module. A controller controls a photographing operation including the exposure of the photographing module and the lighting intensity of the background lighting module. An image processing module processes the images photographed by the photographing module. The controller sets the exposure and lighting intensity to specific conditions so as to photograph a processing image where the background portion on the periphery of the object has a higher luminance than that of the silhouette portion of the object. The image processing module extracts the shape of the object by using the luminance difference between the silhouette area of the object and the background area on the periphery of the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. application Ser. No.10/218,059, filed Aug. 13, 2002. This application is further based uponand claims the benefit of priority from the prior Japanese PatentApplications No. 2001-245593, filed Aug. 13, 2001; and No. 2002-32596,filed Feb. 8, 2002, the entire contents of both of which areincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to a shape extraction system and a 3-D (threedimension) information acquisition system using the shape extractionsystem, and more particularly to a shape extraction system whichextracts the boundary of an object on the basis of a photographic imageobtained in a state where background lighting is applied to the objectwhose two-dimensional image boundary is to be extracted and aphotographic image obtained in a state where no background lighting isapplied to the object and a 3-D information acquisition system whichacquires 3-D information about the object by using the shape extractionsystem as component techniques.

2. Description of the Related Art

As for a conventional shape extraction system, Jpn. Pat. Appln. KOKAIPublication No. 11-73491 has disclosed an image cutout method usinginfrared rays as light of a specific wavelength outside the visiblelight region.

As another conventional shape extraction system, Jpn. Pat. Appln. KOKAIPublication No. 2000-224410 has disclosed a method of determining acutout area by using a light source with a specific chroma and aspecific lightness behind the object, or chromatic techniques.

As still another conventional shape extraction system, Jpn. Pat. Appln.KOKAI Publication No. 10-124704 has disclosed a 3-D informationacquisition apparatus.

The 3-D information acquisition apparatus calculates a hypotheticalexistence area using the boundary between the object and the backgroundin the image.

The hypothetical existence area is a conical area which has theprojection center of the camera at the vertex and the shape of whosecross section is the boundary between the object and the background inthe image.

The conical area (or hypothetical existence area) is written using aboxel (a model expressed by cubes of a specific size).

The above-described process is carried out repeatedly, as the object isturned through a specific angle by a turntable.

Then, a common hypothetical existence area is determined and 3-Dinformation about the object is acquired.

The boundary between the object and the boundary in the image isdetermined from the difference between the image obtained byphotographing only the background in the absence of the object and theimage obtained by photographing the background in the presence of theobject.

Since the image cutout method disclosed in Jpn. Pat. Appln. KOKAIPublication No. 11-73491 cuts out the silhouette of an image usinginfrared rays, the result of cutout might be influenced by theenvironmental condition, particularly the ambient temperature or thetemperature of the object.

Furthermore, the image cutout method requires not only a special lightsource that emits infrared rays but also a special band-pass filter anda special photographing apparatus, resulting in the disadvantage ofmaking the apparatus expansive.

On the other hand, in the cutout area determining method disclosed inJpn. Pat. Appln. KOKAI Publication No. 2000-224410 is at a disadvantagein that it is difficult to cut out accurately an object whose chroma issimilar to that of the background.

Particularly in the cutout area determining method, when a part of theobject is a metal surface (a flat metal surface or a scattering surfacesimilar to this or a part with a relatively high reflectivity), themetal surface part looks the same color as that of the background as aresult of reflecting the background (color). This causes the problem ofthe shape of the object to be recognized erroneously.

In the prior art 3-D information acquisition apparatus disclosed in Jpn.Pat. Appln. KOKAI Publication No. 10-124704, to improve the accuracy of3-D information, it is necessary to make the specific angle smaller.This increases the number of shootings, causing the problem ofincreasing the photographing time.

In the 3-D information acquisition apparatus, to set the photographingangle first, the turntable turns and stops repeatedly. This appliesacceleration to the object, which might cause the object to overturn orbe deformed.

Furthermore, since the stop position of the 3-D information acquisitionapparatus must be controlled with high accuracy, this leads to thedisadvantages that the moving unit and control unit for the turntableare complex and expensive.

In the 3-D information acquisition apparatus, to determine the boundarybetween the object and the background in the image, the differencebetween the image of only the background and the image of the objectwith the background is used. Because the camera exposure, focus, andshutter speed vary from one shooting to another, this causes the problemthat the boundary might not be determined with high accuracy.

Furthermore, in the 3-D information acquisition apparatus, since acommon hypothetical existence area is determined directly by using aboxel model, the number of images necessary to determine whether acertain boxel is included in the common hypothetical existence area islarge, which leads to the disadvantage that the processing time is verylong.

Moreover, in the 3-D information acquisition apparatus, since it isnecessary to store the probability of existence related to each of theboxels in the previously set boxel model, this results in thedisadvantage that the memory capacity must be very high.

BRIEF SUMMARY OF THE INVENTION

An object of the present invention is to provide a shape extractionapparatus and method which are capable of cutting out an image reliablyat low cost and a shape extraction system including an image cutoutapparatus and method, and more particularly a shape extraction systemusing the technique for extracting the boundary of an object on thebasis of a photographic image obtained in a state where backgroundlighting is applied to the object whose two-dimensional image boundaryis to be extracted and a photographic image obtained in a state where nobackground lightning is applied to the object.

Another object of the present invention is to provide a 3-D informationacquisition apparatus and method which are capable of not onlydetermining the boundary with high accuracy and acquiring high-accuracy3-D information but also reducing remarkably the memory capacity toacquire 3-D information about an object, shortening the photographingtime, and keeping the object stable, and a 3-D information acquisitionsystem including a 3-D information acquisition program, and moreparticularly a 3-D information acquisition system which acquires 3-Dinformation about the object by using as component techniques a shapeextraction system which extracts the two-dimensional boundary of theobject on the basis of a photographic image obtained in a state wherebackground lighting is applied to the object whose two-dimensional imageboundary is to be extracted and a photographic image obtained in a statewhere no background lightning is applied to the object.

According to a first aspect of the present invention, there is provideda shape extraction apparatus comprising: a photographing module whichphotographs an object; a background lighting module which illuminatesthe object from behind by visible light with respect to thephotographing module to identify an area including the boundary betweenthe object and the background portion in an image to be photographed bythe photographing module; a control module which controls aphotographing operation including the exposure of the photographingmodule and the lighting intensity of the background lighting module; andan image processing module which processes the image photographed by thephotographing module, wherein the control module sets the exposure andlighting intensity to specific conditions so as to photograph aprocessing image where the background portion on the periphery of theobject has a higher luminance than that of the silhouette portion of theobject, and the image processing module extracts the shape of the objectby using the luminance difference between the silhouette area of theobject and the background area on the periphery of the object in theprocessing image.

According to a second aspect of the present invention, there is provideda shape extraction apparatus according to the first aspect, wherein thebackground lighting module includes at least a light-source module whichemits light in the visible light region and a light scattering modulewhich is provided behind the object with respect to the photographingmodule and scatters light from the light-source module.

According to a third aspect of the present invention, there is provideda shape extraction apparatus according to the second aspect, wherein ascattered light radiation area where the light scattering moduleradiates scattered light includes an area corresponding to the boundarybetween the object and the background portion in the image photographedby the photographing module.

According to a fourth aspect of the present invention, there is provideda shape extraction apparatus according to the first aspect, wherein theprocessing image is a silhouette image photographed in such a mannerthat the object is darker than the background.

According to a fifth aspect of the present invention, there is provideda shape extraction apparatus according to the first aspect, furthercomprising an object placing module on which the object is placed andwhich transmits light from the background lighting module.

According to a sixth aspect of the present invention, there is provideda shape extraction method comprising: photographing an object;illuminating the object from behind by visible light to identify an areaincluding the boundary between the object and the background portion inan image to be photographed; controlling a photographing operationincluding the exposure in photographing the object and the lightingintensity of the lighting; and processing the photographed image,wherein the control sets the exposure and lighting intensity to specificconditions so as to photograph a processing image where the backgroundportion on the periphery of the object has a higher luminance than thatof the silhouette portion of the object, and the processing extracts theshape of the object by using the luminance difference between thesilhouette area of the object and the background area on the peripheryof the object in the processing image.

According to a seventh aspect of the present invention, there isprovided an image cutout apparatus comprising: a photographing modulewhich photographs an object; a background lighting module whichilluminates the object from behind by visible light with respect to thephotographing module to identify an area including the boundary betweenthe object and the background portion in an image to be photographed bythe photographing module; a control module which controls aphotographing operation including the exposure of the photographingmodule and the lighting intensity of the background lighting module; animage processing module which processes the image photographed by thephotographing module; and a storage module which stores the imageprocessed by the image processing module, wherein the control moduleacquires a first image obtained by setting the photographing module to afirst specific exposure and photographing the object, and a second imageobtained by lighting the background lighting module at a specificlighting intensity, setting the photographing module to the firstspecific exposure or a second specific exposure different from the firstspecific exposure, and photographing the object, the image processingmodule extracts the shape of the object by using the luminancedifference between the silhouette portion of the object and thebackground portion on the periphery of the object in the second imageand creates a third image by cutting out an image of a partcorresponding to the object from the first image on the basis of theextracted shape, and the storage module stores the third image.

According to an eighth aspect of the present invention, there isprovided an image cutout apparatus according to the seventh aspect,further comprising a normal lighting module which illuminates the sideto be photographed of a subject, wherein the first image is photographedwith lighting by the normal lighting module.

According to a ninth aspect of the present invention, there is providedan image cutout apparatus according to the seventh or eighth aspect,wherein the background lighting module includes at least a light sourcemodule which emits light in the visible light region and a lightscattering module which is provided behind the object with respect tothe photographing module and scatters light from the light sourcemodule.

According to a tenth aspect of the present invention, there is providedan image cutout apparatus according to the seventh or eighth aspect,wherein a scattered light radiation area where the light scatteringmodule radiates scattered light includes an area corresponding to theboundary between the object and the background portion in the imagephotographed by the photographing module.

According to an eleventh aspect of the present invention, there isprovided an image cutout apparatus according to the seventh or eighthaspect, wherein the second image is such that the background portion onthe periphery of the object has a higher luminance than that of thesilhouette portion of the object.

According to a twelfth aspect of the present invention, there isprovided an image cutout apparatus according to the seventh or eighthaspect, wherein the second image is a silhouette image of the object.

According to a thirteenth aspect of the present invention, there isprovided an image cutout apparatus according to the seventh or eighthaspect, further comprising an object placing module on which the objectis placed and which transmits light from the background lighting module.

According to a fourteenth aspect of the present invention, there isprovided a 3-D information acquisition apparatus comprising: aphotographing module which photograph an image of an object; a relativemovement module which moves the object and the photographing modulerelatively and continuously in such a manner that the photographingmodule can photograph images of the object from a plurality ofviewpoints; a relative position sensing module which senses the relativeposition of the object and the photographing module at each of theviewpoints at which images of the object are photographed by thephotographing module from a plurality of viewpoints; and a 3-D shapeestimating module which estimates a 3-D shape of the object by using theimages of the object photographed by the photographing module from aplurality of viewpoint and information about the relative positionsensed by the relative position sensing module.

According to a fifteenth aspect of the present invention, there isprovided a 3-D information acquisition apparatus comprising: aphotographing module which photograph an image of an object; abackground module which has a specific optical characteristic and whichis provided behind the object and becomes the background of the objectin photography; a relative movement module which moves the object andthe photographing module relatively and continuously in such a mannerthat the photographing module can photograph images of the object from aplurality of viewpoints; a relative position sensing module which sensesthe relative position of the object and the photographing module at eachof the viewpoints at which images of the object are photographed by thephotographing module from a plurality of viewpoints; and a 3-D shapeestimating module which recognizes the areas occupied by the object inthe images photographed by the photographing module from the pluralityof viewpoints by using the images of the object photographed by thephotographing module from the plurality of viewpoints and informationabout the relative position sensed by the relative position sensingmodule and which estimates a 3-D shape of the object by using the areasoccupied by the object.

According to a sixteenth aspect of the present invention, there isprovided a 3-D information acquisition apparatus comprising: aphotographing module which photograph an image of an object; abackground lighting module which illuminates, directly or indirectlyfrom behind the object, a range including at least all of the silhouetteportion of the object or a part of the silhouette portion in thephotographing range of the photographing module; a relative movementmodule which moves the object and the photographing module relativelyand continuously in such a manner that the photographing module canphotograph images of the object from a plurality of viewpoints; arelative position sensing module which senses the relative position ofthe object and the photographing module at each of the viewpoints atwhich images of the object are photographed by the photographing modulefrom a plurality of viewpoints; and a 3-D shape estimating module whichrecognizes an area occupied by the object in each of the imagesphotographed by the photographing module from the plurality ofviewpoints by using the images of the object photographed by thephotographing module from the plurality of viewpoints and informationabout the relative position sensed by the relative position sensingmodule and which estimates a 3-D shape of the object by using the areasoccupied by the object, wherein the background lighting module is turnedon in photographing an image used to recognize the area occupied by theobject.

According to a seventeenth aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to thefifteenth or sixteenth aspect, wherein the relative movement modulerotates the object.

According to an eighteenth aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to theseventeenth aspect, wherein the relative movement module rotates theobject almost at a constant angular speed.

According to a nineteenth aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to theseventeenth aspect, wherein the relative movement module moves theobject in a direction parallel with the axis of rotation.

According to a twentieth aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to theseventeenth, wherein the relative movement module rotates the object ata constant angular speed, and the relative position sensing moduleincludes a reference angular position sensing module which senses areference angular position, a time difference computing module whichcalculates the time difference between the time the reference angularposition sensing module senses a reference angular position and the timethe photographing module photographs, and an angle differencedetermining module which determines the angle difference between thereference angle and the angle through which a turn is made until thephotographing module photographs by using the constant angular speed andthe result obtained at the time difference computing module.

According to a twenty-first aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to thesixteenth aspect, wherein the image obtained by the 3-D shape estimatingmodule in photographing to estimate a 3-D shape of the object is asilhouette image where the area occupied by the object is darker thanthe background portion near the silhouette of the object.

According to a twenty-second aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to thetwenty-first aspect, wherein the 3-D shape estimating module extractsthe silhouette of the object by using the luminance difference betweenthe silhouette area of the object and the background area on theperiphery of the object in the silhouette image and estimates the areaoccupied by the object by using the extracted silhouette.

According to a twenty-third aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to thesixteenth aspect, wherein the background lighting module includes atleast a light source module which emits light in the visible lightregion, and a light scattering module which is provided behind theobject with respect to the photographing module and scatters light fromthe light source module.

According to a twenty-fourth aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to thesixteenth aspect, wherein the 3-D shape estimating module estimates thetexture of the surface of the object by using a texture image of theobject photographed at least once by the photographing module in a statewhere the background lighting module does not illuminate the object.

According to a twenty-fifth aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to thefifteenth or sixteenth aspect, wherein the 3-D shape estimating moduleincludes a closed area setting module which sets in the object a set ofclosed areas arranged closely in a three-dimensional space, and aclosed-area outside determining module which determines whether theclosed area exists outside the object by calculating the probabilitythat each of the closed areas will exist outside the object on the basisof the images obtained by photographing the object by the photographingmodule from a plurality of viewpoints, and the closed area is removedfrom the subsequent closed areas to be determined at the closed-areaoutside determining module, when the closed-area outside determiningmodule determines that the probability that the closed area will existoutside the object exceeds a specific threshold value.

According to a twenty-sixth aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to thefifteenth or sixteenth aspect, wherein the 3-D shape estimating modulecarries out a recognition and estimation process of not only recognizingthe area occupied by the object on the basis of the image photographedwith a visual line from a first viewpoint among the images photographedby the photographing module from a plurality of viewpoints but alsoestimating a 3-D shape of the object, and then carries out a recognitionand estimation process of not only recognizing the area occupied by theobject on the basis of the image photographed with a visual line from asecond viewpoint farthest from the first viewpoint but also estimatingthe 3-D shape of the object, and thereafter repeats the recognition andestimation process using the image from the viewpoint closest to theangle in which the angle difference is interpolated among the remainingones of the viewpoints corresponding to the image not used in therecognition and estimation process and being sandwiched between visuallines producing the largest angle difference corresponding to the twoimages used in the recognition and estimation process.

According to a seventeenth aspect of the present invention, there isprovided a 3-D information acquisition apparatus according to thefifteenth or sixteenth aspect, wherein the 3-D shape estimating moduleincludes a boundary closed area determining module which determineswhether the probability that a closed area belonging to the set ofclosed areas will exist outside the object reaches a specificprobability range and whether the closed area exists near the boundarybetween the inside and outside of the object, and a boundary closed areadividing module which divides the closed area determined to be a closedarea existing near the boundary by the boundary closed area determiningmodule, into subdivision closed areas, and causes the boundary closedarea determining module to determine further whether the subdivisionclosed areas divided by the boundary closed area dividing module existnear the boundary, causes the boundary closed area dividing module tosubdivide the subdivision closed areas on the basis of the result of thedetermination, and repeats the determination by the boundary closed areadetermining module and the division by the boundary closed area dividingmodule until the closed areas have a specific size.

According to a twenty-eighth aspect of the present invention, there isprovided a 3-D information acquisition method of using the imagesobtained by photographing an object from a plurality of viewpoints andinformation about the positions of the viewpoints to recognize the areasoccupied by the object in the images, estimating a 3-D shape of theobject on the basis of the areas occupied by the object, and acquiring3-D information about the object, the 3-D information acquisition methodcomprising: setting in the object a set of closed areas arranged closelyin a three-dimensional space; determining whether the closed area existsoutside the object by finding the probability that each of the closedareas will exist outside the object on the basis of the images obtainedby photographing the object from a plurality of viewpoints; and removingthe closed area from the remaining ones of the closed areas to bedetermined as to whether they exist outside the object, when determiningthat the probability that the closed area will exist outside the objectexceeds a specific threshold value.

According to a twenty-ninth aspect of the present invention, there isprovided a 3-D information acquisition method of using the imagesobtained by photographing an object from a plurality of viewpoints andinformation about the positions of the viewpoints to recognize the areasoccupied by the object in the images, estimating a 3-D shape of theobject on the basis of the areas occupied by the object, and acquiring3-D information about the object, the 3-D information acquisition methodcomprising: carrying out a first recognition and estimation process ofnot only recognizing the area occupied by the object on the basis of theimage photographed with a visual line from a first viewpoint among theimages photographed from the plurality of viewpoints but also estimatinga 3-D shape of the object; carrying out a second recognition andestimation process of not only recognizing the area occupied by theobject on the basis of the image photographed with a visual line from asecond viewpoint farthest from the first viewpoint but also estimatingthe 3-D shape; and thereafter carrying out a third recognition andestimation process similar to the first and second recognition andestimation processes by using the image from the viewpoint closest tothe angle in which the angle difference is interpolated among theremaining ones of the viewpoints corresponding to the images not used inthe first and second recognition and estimation processes and beingsandwiched between visual lines producing the largest angle differencecorresponding to the two images used in the first and second recognitionand estimation processes; and carrying out a fourth recognition andestimation process of repeating the third recognition and estimationprocess.

According to a thirtieth aspect of the present invention, there isprovided a 3-D information acquisition method of using the imagesobtained by photographing an object from a plurality of viewpoints andinformation about the positions of the viewpoints to recognize the areasoccupied by the object in the images, estimating a 3-D shape of theobject on the basis of the areas occupied by the object, and acquiring3-D information about the object, the 3-D information acquisition methodcomprising: setting in the object a set of closed areas arranged closelyin a three-dimensional space; determining whether the probability that aclosed area belonging to the set of closed areas will exist outside theobject reaches a specific probability range and whether the closed areaexists near the boundary between the inside and outside of the object;and dividing the closed area determined to be a closed area existingnear the boundary into subdivision closed areas, wherein the subdivisionclosed areas are subjected to the process of determining whether theyare closed areas existing near the boundary and the process of furtherdividing the closed areas into subdivision closed areas, until theclosed areas have a specific size.

According to a thirty-first aspect of the present invention, there isprovided a 3-D information acquisition program which causes a computerto use the images obtained by photographing an object from a pluralityof viewpoints and information about the positions of the viewpoints torecognize the areas occupied by the object in the images, estimate a 3-Dshape of the object on the basis of the areas occupied by the object,and acquire 3-D information about the object, the 3-D informationacquisition program comprising: a closed-area setting step of causingthe computer to set in the object a set of closed areas arranged closelyin a three-dimensional space; a closed-area outside determining step ofcausing the computer to determine whether the closed area exists outsidethe object by finding the probability that each of the closed areas willexist outside the object on the basis of the images obtained byphotographing the object from a plurality of viewpoints, and a step ofcausing the computer to remove the closed area from the remaining onesof the closed areas to be subjected to the closed-area outsidedetermining step, when the closed-area outside determining stepdetermines that the probability that the closed area will exist outsidethe object exceeds a specific threshold value.

According to a thirty-second aspect of the present invention, there isprovided a 3-D information acquisition program which causes a computerto use the images obtained by photographing an object from a pluralityof viewpoints and information about the positions of the viewpoints torecognize the areas occupied by the object in the images, estimate a 3-Dshape of the object on the basis of the areas occupied by the object,and acquire 3-D information about the object, the 3-D informationacquisition program comprising: a first recognition and estimationprocessing step of causing the computer to not only recognize the areaoccupied by the object on the basis of the image photographed with avisual line from a first viewpoint among the images photographed fromthe plurality of viewpoints but also estimate a 3-D shape of the object;a second recognition and estimation processing step of causing thecomputer to not only recognize the area occupied by the object on thebasis of the image photographed with a visual line from a secondviewpoint farthest from the first viewpoint but also estimate a 3-Dshape of the object; a third recognition and estimation processing stepof causing the computer to carry out the recognition and estimationprocess by using the image from the viewpoint closest to the angle inwhich the angle difference is interpolated among the remaining ones ofthe viewpoints corresponding to the image not used in the first andsecond recognition and estimation processes and being sandwiched betweenvisual lines producing the largest angle difference corresponding to thetwo images used in the first and second recognition and estimationprocesses; and a fourth recognition and estimation processing step ofcausing the computer to carry out the third recognition and estimationprocessing step repeatedly.

According to a thirty-third aspect of the present invention, there isprovided a 3-D information acquisition program which causes a computerto use the images obtained by photographing an object from a pluralityof viewpoints and information about the positions of the viewpoints torecognize the areas occupied by the object in the images, estimate a 3-Dshape of the object on the basis of the areas occupied by the object,and acquire 3-D information about the object, the 3-D informationacquisition program comprising: a closed area setting step of causingthe computer to set in the object a set of closed areas arranged closelyin a three-dimensional space; a boundary closed area determining step ofcausing the computer to determine whether the probability that a closedarea belonging to the set of closed areas will exist outside the objectreaches a specific probability range and whether the closed area existsnear the boundary between the inside and outside of the object; aboundary closed area dividing step of causing the computer to divide theclosed area determined to be a closed area existing near the boundary inthe closed area determining step, into subdivision closed areas; and astep of causing the computer to subject the subdivision closed areas tothe boundary closed area determining step and the boundary closed areadividing step, until the closed areas have a specific size.

According to a thirty-fourth aspect of the present invention, there isprovided a 3-D information acquisition apparatus comprising: aphotographing module which photographs an image of an object; a relativemovement module which moves the object and the photographing modulerelatively in such a manner that the photographing module can photographimages of the object from a plurality of viewpoints; a photographingcontrol module which gives not only a photographing instruction to thephotographing module but also a moving instruction to the relativemovement module; a relative position sensing module which senses therelative position of the object and the photographing module at each ofthe viewpoints at which the object is photographed by the photographingmodule from a plurality of viewpoints, on the basis of the signal fromthe photographing module; and a 3-D shape estimating module whichestimates a 3-D shape of the object by using the images of the objectphotographed by the photographing module from a plurality of viewpointsand information about the relative position sensed by the relativeposition sensing module.

Additional objects and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or may be learned by practice of the invention. The objectsand advantages of the invention may be realized and obtained by means ofthe instrumentalities and combinations particularly pointed outhereinafter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate presently preferred embodiments ofthe invention, and together with the general description given above andthe detailed description of the preferred embodiments given below, serveto explain the principles of the invention.

FIG. 1 shows a basic configuration of a shape extraction systemaccording to a first embodiment of the present invention;

FIG. 2 shows the relationship between the luminance time of a surfacelight-emitting light source and the ambient light source in the shapeextraction system of the first embodiment;

FIG. 3 shows the relationship between the luminance time of a surfacelight-emitting light source and the ambient light source in the shapeextraction system of the first embodiment;

FIG. 4 shows the relationship between the luminance time of a surfacelight-emitting light source and the ambient light source in the shapeextraction system of the first embodiment;

FIGS. 5A to 5C are diagrams to help explain a background light modelingby curve fitting related to the shape extraction system of the firstembodiment and a cutout method using the modeling;

FIGS. 6A to 6E show a method of creating a light source modeling in theshape extraction system of the first embodiment;

FIG. 7 is drawings to help explain the process of cutting out an imagein the shape extraction system of the first embodiment;

FIG. 8 shows a case where a transparent photographic stand and a flashare used in the shape extraction system of the first embodiment;

FIG. 9 shows a case where a camera in the shape extraction system of thefirst embodiment is provided with a subject lighting flash;

FIG. 10 is a drawing to help explain a field angle in a case where thecamera in the shape extraction system of the first embodiment isprovided with the subject lighting flash;

FIG. 11 shows a basic configuration of a shape extraction systemaccording to a second embodiment of the present invention;

FIG. 12 shows a case where a camera in the shape extraction system ofthe second embodiment is provided with two subject lighting flashes;

FIG. 13 shows a case where a background lighting light source in theshape extraction system of the second embodiment is placed in front of ascattering reflector;

FIG. 14 shows a case where a backlight panel in the shape extractionsystem of the second embodiment is provided behind a scattering screen;

FIG. 15 shows a case where a front light panel in the shape extractionsystem of the second embodiment is provided in front of the scatteringscreen;

FIG. 16 shows a basic configuration of a shape extraction systemaccording to a ninth embodiment of the present invention;

FIGS. 17A and 17B are drawings to help explain the effect of apolarizing filter applied to the shape extraction system of the ninthembodiment;

FIG. 18 shows a basic configuration of a shape extraction systemaccording to a tenth embodiment of the present invention;

FIG. 19 is a diagram to help explain a photographic image profile in theshape extraction system of the present invention;

FIG. 20 shows a basic configuration of a shape extraction systemaccording to an eleventh embodiment of the present invention;

FIG. 21 is a block diagram showing the configuration of a 3-Dinformation acquisition system according to a twelfth embodiment of thepresent invention;

FIG. 22 is a flowchart showing a general processing flow in the 3-Dinformation acquisition system of the twelfth embodiment;

FIG. 23 shows the relationship between a camera coordinate system and animage coordinate system applied to the 3-D information acquisitionsystem of the twelfth embodiment;

FIGS. 24A and 24B show the relationship between a camera coordinatesystem Oc and a turntable coordinate system Or set on a turntable RU inthe 3-D information acquisition system of the twelfth embodiment;

FIG. 25 shows a pattern flat plate used as an example of determining arotation matrix Rcr and a translation vector Tcr applied to the 3-Dinformation acquisition system of the twelfth embodiment;

FIG. 26 is a drawing to help explain a method of photographing with acamera from a plurality of different angles obtained by standing thepattern flat plate of FIG. 25 straight on the turntable and rotating theturntable in steps of 10 degrees, as an example of determining arotation matrix Rcr and a translation vector Tcr applied to the 3-Dinformation acquisition system of the twelfth embodiment;

FIG. 27 shows object images A01, A02, . . . , A36 applied to the 3-Dinformation acquisition system of the twelfth embodiment;

FIG. 28 shows boundary images B01, B02, . . . , B36 applied to the 3-Dinformation acquisition system of the twelfth embodiment;

FIG. 29 is a flowchart to help explain the flow of the process in stepS3 of FIG. 22;

FIG. 30 shows a boxel BOX applied to the twelfth embodiment;

FIGS. 31A and 31B show an example of external determination applied tothe twelfth embodiment;

FIG. 32 is a drawing to help explain how a boxel is determined to beexternal with boundary image B01 applied to the 3-D informationacquisition system of the twelfth embodiment;

FIGS. 33A to 33D are drawings to help explain how an object applied tothe 3-D information acquisition system of the twelfth embodiment is cutout, using a two-dimensional simple shape;

FIG. 34 is a block diagram showing the configuration of a modificationof the 3-D information acquisition system according to the twelfthembodiment;

FIG. 35 is a flowchart to help explain the flow of processing in a 3-Dinformation acquisition system according to a thirteenth embodiment ofthe present invention;

FIG. 36 is a block diagram showing the configuration of a 3-Dinformation acquisition system according to a fourteenth embodiment ofthe present invention;

FIG. 37 shows silhouette images S01, S02, . . . , S36 applied to the 3-Dinformation acquisition system of the fourteenth embodiment;

FIG. 38 is a block diagram showing the configuration of a 3-Dinformation acquisition system according to a fifteenth embodiment ofthe present invention;

FIG. 39 shows photographic images applied to the 3-D informationacquisition system of the fifteenth embodiment;

FIGS. 40A to 40D are drawings to help explain how a boxel changes in a3-D information acquisition system according to a sixteenth embodimentof the present invention;

FIG. 41 is a block diagram showing the configuration of a 3-Dinformation acquisition system according to a seventeenth embodiment ofthe present invention;

FIG. 42 is a block diagram showing the configuration of a modificationof the 3-D information acquisition system according to the seventeenthembodiment;

FIG. 43 is a block diagram showing the configuration of a final productof a shape extraction system including 3-D information to which thepresent invention is applied; and

FIG. 44 is drawings to help explain an application to which theconfiguration of FIG. 41 is applied.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the presently preferredembodiments of the invention as illustrated in the accompanyingdrawings, in which like reference numerals designate like orcorresponding parts.

First Embodiment

FIG. 1 shows a basic configuration of a shape extraction systemaccording to a first embodiment of the present invention;

As shown in FIG. 1, in the shape extraction system of the firstembodiment, a surface light-emitting light source 13 serving as abackground lighting module is provided behind an object 10 to bephotographed.

The surface light-emitting light source 13 is connected to aphotographic condition control unit 12.

The photographic condition control unit 12 is connected to a camera 11located in front of the object 10.

The camera 11 is connected via an image processing unit 14 to an outputunit 15.

The surface light-emitting light source 13 emits visible scattered lightand illuminates the whole of the object 10 from behind.

The photographic condition control unit 12 controls the photographingoperation of the camera 11 and its exposure condition and thelight-emitting operation and light-emitting intensity of the surfacelight-emitting light source 13.

In a state where a specific condition is set and the surfacelight-emitting light source 13 is turned on, that is, in a state wherebackground lighting is applied to the object 10, the photographiccondition control unit 12 performs such control as acquires a firstimage (silhouette image) obtained by photographing the object 10 withthe camera 11.

Furthermore, in a state where a specific condition is set and thesurface light-emitting light source 13 is turned off, that is, in astate where no background lighting is applied to the object 10, thephotographic condition control unit 12 performs such control as acquiresa second image obtained by photographing the object 10 with the camera11.

In the shape extraction system of the present invention, the shape ofthe object is extracted on the basis of the first image (silhouetteimage) and second image obtained as described above.

That is, in the present invention, the shape of the object 10 isrecognized on the basis of the first image (silhouette image) and acutout mask is formed. Using this mask, an object area is cut out fromthe second image.

The photographic condition (the relationship between the luminance ofthe light source and the exposure of the camera) for acquiring the firstimage (silhouette image) will be explained by reference to FIGS. 2 to 4.

A flash in FIGS. 2 to 4 means light emitted from the surfacelight-emitting light source 13. In the first embodiment, it is assumedthat the photographic condition control unit 12 causes the surfacelight-emitting light source 13 to emit light with a specific intensityfor a specific time.

Ambient light in FIGS. 2 to 4 means light from the lighting light sourcein the room, for example, when photographing is done in a room.

FIG. 2 shows a case where a flash is higher than the ambient light in acase where photographing is done with the camera being set to a specificexposure by the photographic condition control unit 12.

In this case, a preferable first image (silhouette image) for cuttingout the shape of the object 10 is obtained.

FIG. 3 shows a case where the shutter speed of the camera 11 is madeslower according to the specific exposure condition of FIG. 2, therebymaking the flash almost equal to the ambient light.

FIG. 4 shows a case where the exposure time of FIG. 3 is made muchslower and the flash is lower than the ambient light.

Although it is not that the shape of the object 10 cannot be cut out atall under the conditions of FIGS. 3 and 4, they are undesirable toobtain a preferable first image (silhouette image) for cutting out theshape of the object 10.

To pick up a first image (silhouette image) preferable for cutting outthe shape of the object 10, the photographic condition control unit 12sets the exposure of the camera 11 and the tuning on of the surfacelight-emitting light source 13 according to the condition of FIG. 2.

It goes without saying that the light from the surface light-emittinglight source 13 is controlled so as not to permit overexposure to causeflares or ghosts in the pickup image.

By setting the above photographic conditions, a silhouette image isobtained. To recognize the shape of the object 10 from the silhouetteimage, the image processing unit 14 binarizes the image.

Using FIGS. 5A, 5B, and 5C, a binarized image acquisition method in thefirst embodiment will be explained.

FIGS. 5A, 5B, and 5C are diagrams to help explain a background lightmodeling by curve fitting and a cutout method using the background lightmodeling.

First, the distribution of background lighting light is measured asshown in FIG. 5A.

For example, as shown in FIG. 5B, binarization is performed with aspecific threshold value, regardless of the object and the backgroundcurve. Therefore, if the contrast between the object and the backgroundis insufficient, the object cannot be cut out.

In the first embodiment, however, as shown in FIG. 5C, use of abackground light modeling by curve fitting that performs a binarizationprocess using a threshold value that draws a parabolic curve makes itpossible to cut out even an object that is not contrasted with thebackground, to a certain extent.

FIGS. 6A to 6E are drawings to help explain a method of creatinglight-source modeling.

FIGS. 6A to 6C are diagrams to help explain a method of photographing animage without an object.

FIGS. 6D and 6E are diagrams to help explain a method of photographingan image when background light modeling is performed using an image withan object.

First, as shown in FIG. 6A, photographing is done with no object in thepresence of a flash (e.g., a surface light-emitting light source).

Next, as shown in FIG. 6B, an object to be photographed is placed andphotographed with the camera 11 in the presence of the flash.

Finally, as shown in FIG. 6C, the object to be photographed is placedand is photographed with the camera 11 in the absence of the flash.

The details of subsequent processes, including image cutout, will beexplained later.

In the method of photographing an image when an image with no object isnot used for background light modeling, the process of FIG. 6A is notused and only the processes of FIGS. 6D and 6E similar to those of FIGS.6B and 6C are carried out.

On the assumption that an image is positioned on the x-axis and they-axis, the intensity f(x, y) is determined using the followingequation:f(x,y)=ax ² +by ² +cx+dy+exy+f

where parameters a, b, c, d, e, f are determined using, for example, aparameter estimation method, such as a method of least squares.

In this example, although quadratic surface approximation is used inlight source modeling, for example, a Nurb curve or a linear model maybe used in the modeling.

In image processing means, if f(x, y) is used as a light source modeland I(x, y) is the pixel value of a silhouette (a photographing imagewhere the object photographed with a light source is dark), and T(0to 1) is a threshold value, cutout can be performed by using thefollowing expression:{f(x,y)−I(x,y)}/f(x,y)>T

FIG. 7 is drawings to help explain an image cutout process.

First, photographing is done with the camera 11, while the surfacelight-emitting light source 13 is emitting light.

At this time, the object becomes a dark image (or silhouette image) asshown in step 21.

Next, another image is photographed with the camera 11 in a state wherethe surface light-emitting light source 13 is prevented from emittinglight.

At this time, the photographic image is the same as a normallyphotographed image. A color photographed image is represented in color.

Then, an image (silhouette image) photographed in a state where thesurface light-emitting light source 13 is emitting is used to extractthe edge or the boundary in step 22.

The extraction may be realized by the cutout function of an ordinaryimage application program, using the result of the shape extraction.

Next, in step 23, a silhouette of an image, called a mask layer, iscreated.

Then, after the mask layer is reversed in step 24, the image obtained byreversing the mask layer in step S25 and the image photographed in thestate where the surface light-emitting light source 13 is stopped fromemitting light are subjected to the prior art reversed mask layersubtraction.

As a result, the background is removed and the cutout of object iscompleted.

As described above, the first embodiment has the advantages of beingless liable to be affected by the temperature of the object and theambient temperature than a conventional method using infrared rays andof requiring neither a special light source nor a special photographingunit.

Since the conventional method using infrared rays uses a non-visiblelight source, it is difficult to make adjustments in assembly,manufacture, and installation. For example, lighting positionadjustments take a lot time.

In contrast, the first embodiment is based on the assumption that whitevisible light is used, so that it is easy to do the assembly andadjustment, etc.

Furthermore, the first embodiment can cope with a case where thebackground color coincides with the color of the object, which was aproblem in a conventional chromatic method, there is no need to changethe background color in response to the color of the object.

Second Embodiment

FIG. 8 shows a basic configuration of a shape extraction systemaccording to a second embodiment of the present invention.

The second embodiment has the same configuration as that of FIG. 1except that a transparent photographic stand 32 on which the object isplaced and a background lighting flash 33 in addition to the surfacelight-emitting light source 13 are used.

Specifically, in the second embodiment, the object 10 is placed on thetransparent photographic stand 32 as shown in FIG. 8.

This enables the object 10 to be fixed easily. Use of the transparentphotographic stand 32 makes it difficult for the stand 32 to appear onthe silhouette image and enables the stand 32 to be removed as thebackground. That is, unwanted items will not appear on the image (whichwill be explained later).

In the second embodiment, in addition to the surface light-emittinglight source 13, a flash 33 may be used.

The second embodiment can carry out the same processes as those in thefirst embodiment.

Third Embodiment

FIG. 9 shows a basic configuration of a shape extraction systemaccording to a third embodiment of the present invention.

The third embodiment has the same configuration as that of FIG. 8 exceptthat the camera 11 is provided with a subject lighting flash 30.

Specifically, in the third embodiment, use of the subject lighting flash30 assures brightness for an image photographed from the front, whichenables the subject to be photographed more clearly.

FIG. 10 is a drawing to help explain a field angle when the subjectlighting flash 30 is provided.

As shown in FIG. 10, it is desirable that the field angle α of thesubject lighting flash 30 should be such that all of the object 10 fitsin the angle.

It is desirable that the field angle β of the camera 11 should be suchthat all of the object 10 fits in the angle.

Furthermore, it is desirable that the field angle γ of the backgroundlighting flash 33 should be such that all of the back of the object 10fits in the angle, with lighting applied from behind the object 10.

The third embodiment can also carry out the same processes as those inthe first embodiment.

Fourth Embodiment

FIG. 11 shows a basic configuration of a shape extraction systemaccording to a fourth embodiment of the present invention.

The fourth embodiment has the same configuration as that of FIG. 9except that a stand for supporting the object 10 is integrated with abackground screen 16.

The fourth embodiment can also carry out the same processes as those inthe first embodiment.

Fifth Embodiment

FIG. 12 shows a basic configuration of a shape extraction systemaccording to a fifth embodiment of the present invention.

In the configuration of the fifth embodiment, the camera 11 of FIG. 9 isprovided with two subject lighting flashes.

Specifically, in the fifth embodiment, a subject lighting flash 30-1 anda surface light-emitting light source 13-1 are additionally providedabove the object 10 as shown in FIG. 12, which enables the subject to bephotographed from the front according to various scenes.

The number of subject lighting flashes may be three or more.

The fifth embodiment can also carry out the same processes as those inthe first embodiment.

Sixth Embodiment

FIG. 13 shows a basic configuration of a shape extraction systemaccording to a sixth embodiment of the present invention.

In the configuration of the sixth embodiment, a background lightinglight source 40 is provided in front of a scattering reflector 13-2instead of the surface light-emitting light source 13 of FIG. 1.

The sixth embodiment can carry out the same processes as those in thefirst embodiment by causing the background lighting light source 40 toproject light on the front of the scattering reflector 13-2.

Seventh Embodiment

FIG. 14 shows a basic configuration of a shape extraction systemaccording to a seventh embodiment of the present invention.

In the configuration of the seventh embodiment, a backlight panel 41 isprovided behind a scattering screen 13-3 instead of the surfacelight-emitting light source 13 of FIG. 1.

A light source 42 is provided in such a manner that it is in contactwith the backlight panel 41.

The seventh embodiment can carry out the same processes as those in thefirst embodiment by providing the backlight panel 41 behind thescattering screen 13-3.

Eighth Embodiment

FIG. 15 shows a basic configuration of a shape extraction systemaccording to an eighth embodiment of the present invention.

In the configuration of the eighth embodiment, a front light panel 43 isprovided in front of the scattering screen 13-3 instead of the surfacelight-emitting light source 13 of FIG. 1.

The light source 42 is provided in such a manner that it is in contactwith the front light panel 43.

The eighth embodiment can carry out the same processes as those in thefirst embodiment by providing the front light panel 43 in front of thescattering screen 13-3.

Ninth Embodiment

FIG. 16 shows a basic configuration of a shape extraction systemaccording to a ninth embodiment of the present invention.

In the configuration of the ninth embodiment, a polarizing filter 50 anda transparent photographic stand 32 are used in the configuration ofFIG. 9.

Specifically, in the ninth embodiment, the polarizing filter 50 isprovided in front of the lens of the camera 11 as shown in FIG. 16.

The remaining configuration is the same as that of FIG. 9. That is, thesubject lighting flash 30, background lighting flash 33, and transparentphotographic stand 32 are arranged as in FIG. 9.

In the ninth embodiment, use of the polarizing filter 50 prevents areflection as shown in FIG. 17B.

Specifically, when the polarizing filter 50 is not used, the reflection51 of the photographic stand 32 appears in the photographic image asshown in FIG. 17A.

However, providing the polarizing filter 50 in front of the lens of thecamera 11 enables an image 51A with no reflection to be photographed asshown in FIG. 17B.

The same effect can be produced by using an antireflection film or thelike in place of or together with the polarizing filter 50.

The polarizing filter 50 may be provided in front of the surfacelight-emitting light source 13. In addition to this, the polarizingfilter 50 may be provided in an arbitrary place.

The ninth embodiment can also carry out the same processes as those inthe first embodiment.

Tenth Embodiment

FIG. 18 shows a basic configuration of a shape extraction systemaccording to a tenth embodiment of the present invention.

In the configuration of the tenth embodiment, a photographic standrotating unit 54 is provided under the transparent photographic stand 32in the configuration of FIG. 9.

Specifically, in the tenth embodiment, use of the photographic standrotating unit 54 enables photographing to be done, while rotating theobject 10. This makes it possible to photograph the object 10 from aplurality of viewpoints with different angles.

The tenth embodiment can also carry out the same processes as those inthe first embodiment.

Use of a plurality of images enables the object 10 to be representedthree-dimensionally as explained later in an embodiment related to a 3-Dinformation acquisition system.

FIG. 19 is a diagram to help explain a photographic image profile.

Specifically, object gradation is represented in the curve of the screenas shown in FIG. 19. With the background lighting light explained in theabove embodiments, the difference between the object gradation and thescreen background is recognized clearly.

Eleventh Embodiment

FIG. 20 shows a basic configuration of a shape extraction systemaccording to an eleventh embodiment of the present invention.

In the eleventh embodiment, moving pictures are photographed.

Specifically, in the eleventh embodiment, two types of images, normalimages of object illuminated from its front side and silhouette imagesof the object illuminated from its back side, are photographedalternately as shown in FIG. 20.

First, using a flash 60, a normal image is taken with the camera 11. Atthis time, a flash 61 does not emit light.

Next, using a flash 63, a silhouette image is taken with the camera 11.At this time, a flash 62 does not emit light.

At this stage, one image is synthesized. From this time on, the aboveprocesses are repeated sequentially, which enables to cutout the movingobject in moving picture to be taken.

The transparent photographic stand used in the shape extraction systemin each of the above embodiments may be made of, for example, Kuraraymethacrylate resin “Paraglass”®.

In addition, transparent photographic stand used in the shape extractionsystem in each of the above embodiments may be made of, for example,colorless, clear acrylic resin material used for water tanks orwindbreaks, such tempered glass as Asahi Glass “Tempelite”®, or such ahighly transmitting optical glass as is used for prisms and lenses (ex.provided by Ohara).

As for the background screen to be photographed, serving as scatteringmeans, for example, a white mat, such as an OS Kogyo white mat forprojection, or a diffusion screen using white emboss processing andpearl processing are suitable for durable reflection scattering means.

The background screen to be photographed, which is scattering means, maybe a white screen subjected to mat processing or be made of aninexpensive material, such as white Japanese vellum.

Transmission scattering means may be, for example, an OS Kogyo durablesingle element flannel wrench screen which incorporates minute flannellenses and microscopic lenses into an acrylic resin plate, or a cinecoatscreen formed by coating an acrylic resin material with diffusingparticles.

Furthermore, as relatively inexpensive means, the background screen tobe photographed, scattering means, may be a flexible rear screen formedby tempering vinyl chloride resin with diffusing particles and shapingthe resulting resin.

In addition, tracing paper or thin quality paper, which is less durablebut disposable, may be used as the background screen to be photographed.

As described above, according to the shape extraction systems in thefirst to eleventh embodiments, a silhouette image representing an exactshape of the object can be obtained without being affected by suchconditions as the temperature of the object and the object's surfacecharacteristic. Therefore, it is possible to cut out an image of theobject reliably and extract the shape of the object to be photographed,without a special light source or a special photographing unit,regardless of the temperature of the object and the ambient temperature.

Furthermore, according to the shape extraction systems in the first toeleventh embodiments, since the light source used for backgroundlighting is inexpensive, neither a special band-pass filter nor aspecial photographing unit is needed, and the light source forphotographing a subject can be shared, and the flexibility in thephotographing site is high.

An embodiment of a 3-D information acquisition apparatus will beexplained which uses the shape extraction system in each of the first toeleventh embodiments as component techniques.

Although explanation will be given mainly in a case where an object ismoved, the same holds true when the photographing means is moved.

Twelfth Embodiment

FIG. 21 is a block diagram showing the configuration of a 3-Dinformation acquisition apparatus according to a twelfth embodiment ofthe present invention.

As shown in FIG. 21, the twelfth embodiment comprises a camera CU asphotographing means, a flash FU for lighting a subject, a turntable (ora photographic stand rotating unit) RU as relative position moving meansRU, a sensor RU that recognizes that the turntable makes one turn, ablue back BB as a background plate, an object BE, a photographing timerecording unit TM, a relative position computing unit PC as relativeposition determining means, and a shape estimating unit SR thatestimates a shape.

FIG. 22 is a flowchart for general processing in the twelfth embodiment.

As shown in FIG. 22, the flow of general processing in the twelfthembodiment comprises step S0 of making calibration, step S1 ofphotographing, step S2 of creating a boundary image, and step S3 ofestimating the shape.

Step S0 of making calibration has only to be carried out only onceunless the focus position and zoom position of the camera CU and thepositional relationship between the camera CU and the turntable RUremain unchanged.

(Step S0)

In step S0, calibration is made.

Calibration in the twelfth embodiment means the process of determiningthe internal parameters of the camera CU and the positional relationshipbetween the camera CU and the turntable RU to find on which point of thephotographed image a point in a three-dimensional space is projected.

First, the internal parameters of the camera CU will be explained.

The internal parameters of the camera CU are the vertical enlargementratio α_(U) of the photographed image, the horizontal enlargement ratioα_(V), and the optical center (a perpendicular line drawn from theposition of the main point to the surface of the image as shown in FIG.3) u₀, v₀.

The enlargement ratio is the ratio of the width of a pixel on thevertical and horizontal image to the distance between the optical mainpoint of the camera CU and the image surface.

FIG. 23 shows the relationship between the camera coordinate system andthe image coordinate system.

For example, in the image coordinate system Oi, the origin Oi is set atthe top left of the image surface IP, the u-axis is set in thehorizontal direction, and the v-axis is set in the vertical direction.In the camera coordinate system Oc, the origin Oc is set in the positionof the main point of the camera CU, the X-axis is set in parallel withthe u-axis, the Y-axis is set in parallel with the v-axis, and theZ-axis is set in the direction of the image surface.

When a certain point is expressed by

W=(x, y, Z)ˆT on the camera coordinate system Oc and by I=(u, v) ˆT onthe image coordinate system Oi, the relationship between these twocoordinate systems Oc, Oi is as follows:I′=(u′,v′,w′)ˆT=U×W′I=(u′/w′,v′/w′)ˆTW′=(x,y,z,1)ˆT

where ˆT represents the transposition of a vector and U represents atransformation matrix from the camera coordinate system Oc to the imagecoordinate system Oi.

Using the perspective ratio α_(U) in the u direction, the perspectiveratio α_(V) in the v direction, and the optical center u₀, v₀, U isexpressed by: $U = \begin{pmatrix}\alpha_{u} & 0 & u_{0} & 0 \\0 & \alpha_{v} & v_{0} & 0 \\0 & 0 & 1 & 0\end{pmatrix}$

A method of determining these parameters has been disclosed in thereference “Three-dimensional CG made from pictures” by Tuyoshi Jyo,Kindai Kagakusha, 2001.

Next, the positional relationship between the camera CU and theturntable RU will be explained.

This is equivalent to the relationship between the camera coordinatesystem Oc and the turntable coordinate system Or set on the turntableRU.

FIG. 24 shows the relationship between the camera coordinate system Ocand each coordinate system on the turntable coordinate system Or.

In the turntable coordinate Or, the axis of rotation is set in the Zdirection and the rotating plate is set in the XY plane.

When a certain point is expressed by

W=(x, y, z)ˆT on the camera coordinate system Oc and by Z=(p, q, r)ˆT onthe turntable coordinate system Or, the relationship between these twocoordinate systems Oc, Or is as follows:W=M×Z′W′=(x,y,z,1)ˆTZ′=(p,q,r,1)ˆT

Here, using a rotation matrix Rcr expressing the positional relationshipof the turntable coordinate system Or viewed from the camera coordinatesystem Oc and a translation vector Tcr, M is expressed by:$M = \begin{pmatrix}\quad & \quad & | & \quad \\\quad & {Rcr} & | & {Tcr} \\\quad & \quad & | & \quad \\0 & 0 & 0 & 1\end{pmatrix}$

A method of determining a rotation matrix Rcr and a translation vectorTcr is, for example, to stand a pattern plate PB straight on therotating plate of the turntable RU as shown in FIG. 26 and photographingthe plate with the camera CU from a plurality of different angles,rotating in steps of 10 degrees. In the pattern plate PB, pattern PCsare arranged at intervals of d as shown in FIG. 25.

For example, if the plate PB is photographed at five different angles,coordinates PC111, PC112 . . . PC115, PC211, PC212, . . . , PC215,PC311, PC312, . . . , PC315, PC411, PC412, . . . , PC415, PC511, PC512,. . . , PC515 on each image coordinate system Oi of the center ofgravity of all the patterns PC11, PC12, . . . , PC15, PC21, PC22, . . ., PC25, PC31, PC32, . . . , PC35, PC41, PC42, . . . , PC45, PC51, PC52,. . . , PC55 in the photographed images IPB1, IPB2, . . . , IPB5 aredetermined.

Consider a pattern plate coordinate system Op where the Z direction isset in the vertical direction of the pattern plate, the Y direction isset in the horizontal direction, the X direction is set in the normaldirection of the plate, and the origin is set at the bottom left of theplate. In this case, coordinates PC11, PC12, . . . , PC15, PC21, PC22, .. . , PC25, PC31, PC32, . . . , PC35, PC41, PC42, . . . , PC45, PC51,PC52, . . . , PC55 on the pattern plate coordinate system Op of eachpattern is expressed by:PCnm=(0,m×d,n×d)ˆT

Here, if a transformation matrix from the pattern plate coordinatesystem Op to the turntable coordinate system Or is G and the vectorafter the transformation is PCRpnm, this gives:${PCRpnm} = {( {{PCRpnmx},{PCRpnmy},{PCRpnmz}} )\hat{}T}$$\begin{matrix}{{PCRpnm}^{\prime} = {( {{PCRpnmx},{PCRpnmy},{PCRpnmz},1} )\hat{}T}} \\{= {{Rp} \times G \times {PCnm}^{\prime}}}\end{matrix}$${PCnm}^{\prime} = {( {0,{m \times d},{n \times d},1} )^{\hat{}}T}$$G = {{\begin{pmatrix}{\cos\quad\alpha} & {{- \sin}\quad\alpha} &  0 | & t_{rpx} \\{\sin\quad\alpha} & {\cos\quad\alpha} &  0 | & t_{rpy} \\0 & 0 &  0 | & 0 \\0 & 0 & 0 & 1\end{pmatrix}{Rp}^{\prime}} = \begin{pmatrix}\quad & \quad & | & 0 \\{Rp} & \quad & | & 0 \\\quad & \quad & | & 0 \\0 & 0 & 0 & 1\end{pmatrix}}$

where subscript p represents one of the plurality of images and Rp is aknown matrix representing the rotation of the turntable RU.

That is, solving the simultaneous equations Ppnm′=U×M×Rp×G×PCnm′ givesthe relationship between the camera coordinate system Oc and theturntable coordinate system Or, that is, the matrix M.

If the coordinate of point F in a certain three-dimensional space viewedfrom the turntable coordinate system Or is Fr and the coordinate ofpoint F viewed from the image coordinate system Oi is Fi, use of Mgives:Fi′=(u′,v′,w′)ˆT=U×M×Fr′Fi=(u′/w′,v′/w′)ˆTFr′=(FrˆT,1)ˆT

This makes it possible to know on which point an arbitrary point in thethree-dimensional space is projected.

Instead of the pattern flat plate, any three-dimensional shape, such asa cylinder or a quadratic prism may be used, provided that acharacteristic pattern is put on the solid surface of the cylinder,prism, or another special shape and the three-dimensional coordinates ofthe center of gravity of the pattern is determined exactly.

(Step S1)

Next, in step S1, an object BE is placed on the turntable RU. Whilebeing rotated, the object BE, together with the background BB, isphotographed with the camera CU, which produces object images A1, A2, .. . , An.

For example, as the object BE is rotated at a rotational speed of 1 rpm,it is photographed 36 times at almost regular intervals during one turn(that is, it is photographed at intervals of 60/36=about 1.67 seconds).As a result, a plurality of object images A01, A02, . . . , A36 areobtained as shown in FIG. 27.

At this time, the subject lighting flash FU may be operated insynchronization with the photographing.

In this case, if photographing is done at a shutter speed of 1/500, theobject BE is rotated through only 0.012 degrees (=360÷60÷500), with theresult that the image is less liable to move slightly.

When photographing is effected using a flash, the photographing timebecomes shorter by several microseconds to several milliseconds, thereis almost no problem.

The number of revolutions has only to be large enough to allow theobject BE to rotate stably and prevent the image from moving slightly.

Here, times T1, T2, . . . , T36(s) at which object images A01, A02, . .. , A36 were photographed are recorded by the photographing timerecording unit TM.

Times T1, T2, . . . , T36 may be, for example, attached to images orpaired with the photographing sequence and stored as a table in anotherfile.

The turntable RU is provided with a sensor RS so as to recognize therotation of the table RU.

This makes it possible to measure the time required for the turntable RUto make one turn. Calculating the average angular speed of one turndecreases the effect of fluctuations in the angular speed of theturntable RU on the three-dimensional configuration.

As for the background, for example, a blue back background BB is used asshown in FIG. 21.

Anything may be used as the background, provided that it is recognizedas the background. For example, red, yellow, or green may be usedinstead of blue.

A pattern, such as a checked pattern, may also used.

(Step S2)

In step S2, the shape estimating unit SR of FIG. 21 creates boundaryimages from object images A01, A02, . . . , A36.

That is, the background BB is recognized from the object images A01,A02, . . . , A36 and a plurality of boundary images B01, B02, . . . ,B36 are created as shown in FIG. 28.

These are binary images, with an area with an object being “1” and anarea with no object being “0.”

An area where it is impossible to make a distinction between the objectand the background may be stored using a number other than “0” and “1.”

Since the number of object images A01, A02, . . . , A36 is 36, thenumber of boundary images is 36.

(Step S3)

Next, in step S3, the shape estimating unit SR of FIG. 21 estimates theshape.

FIG. 29 is a flowchart for the process in step S3.

First, in step S301 of boxel setting, a boxel BOX is set on theturntable coordinate system Or as shown in FIG. 30.

In this case, the setting range of boxel BOX is set in an area thatcovers the object completely.

The size of a cell BS and the number of cells BS are set according tothe desired accuracy.

For instance, if the setting range is a cube with a diagonal of (−1, −1,0), (1, 1, 2), the size of one cell is 0.001 and the number of cells BSis 2000×2000×2000=8×10ˆ9 where ˆ9 means the ninth power.

In the setting range, a sphere with a diameter of 2 put almost in thecenter of the turntable RU can be measured.

While in the twelfth embodiment, the cells BS and the setting range ofboxel are cubic, they may be, for example, rectangular parallelepiped, atriangular prism, or a hexagonal prism. In a cylindrical coordinatesystem, fan-shaped cells and a fan-shaped boxel may be used.

When the object is flat, the amount of calculations and the memorycapacity can be decreased remarkably by making the boxel setting rangerectangular parallelepiped.

Next, it is determined in step S302 whether all of the images have beenprocessed. If not, one of the unprocessed images is selected in stepS303 of processing image selection.

Then, in step S304 of photographing angle computation, photographingangles Aa1, Aa2, . . . , Aa36 are determined from a reference angle.

This is done in the relative position computing unit PC of FIG. 21.

In this case, the reference angle may be, for example, the angle atwhich photographing was done for the first time or another photographingangle.

In the twelfth embodiment, since the turntable RU is rotated at arotational speed of 1 rpm, the turntable RU rotates six degree in asecond. This gives:Aan=(Tn−T1)×6

In step S305, it is determined whether all of the boxels have beenprocessed. If not, a boxel to be determined is selected in step S306 ofto-be-determined boxel selection.

Then, in step S307, it is determined whether a boxel is outside theobject. Of the boxels not determined to be outside the object, anundetermined boxel is selected.

The selected boxel is projected onto boundary images B01, B02, . . . ,B36 using the transformation matrix and the rotation matrix Rpcorresponding to photographing angles Aa1, Aa2, . . . , Aa36 in stepS308 of boxel vertex coordinate projection. Then, in step 309 of boxeloutside determination, if all of the vertexes are not included in theobject area, it is determined that the boxel is outside the object.

In the outside determination, the center of gravity of the boxel may beprojected on boundary images B01, B02, . . . , B36 and, when theprojected point is not included in the object area, it may be determinedthat the boxel is outside the object.

The result of the determination is stored in the result-of-determinationstorage section.

FIG. 31 shows examples of the determination.

Specifically, FIG. 31A shows a case where it is determined that theboxel is inside the object. FIG. 31B shows a case where it is determinedthat the boxel is outside the object.

FIG. 32 shows how boxels are determined to be outside the object on thebasis of boundary image B01.

In the figure, the shaded portion is the boxels determined to be outsidethe object.

FIGS. 33A to 33D show two-dimensional examples to help explain how theobject is cut out by this method.

In the figures, the shaded portions are the cut-out boxels and thecrosshatched portions are the boxels previously determined to be outsidethe object and therefore not to be determined this time.

A boxel determined to be outside the object the number of times largerthan a threshold value at the boxel outside determining section isregistered as an outside boxel.

The determination may be made using the outside probability=the numberof determinations/the number of images used in the determinations.

Finally, after all of the images have been processed, the boxels notoutside the object are made a 3-D shape of the object.

Photographing may be done in a plurality of turns with the photographingsection.

In this case, photographing at a different viewpoint in each turnproduces the following effect.

For example, when photographing is done with an increasing resolution inthe direction of rotation, for example, in a first turn near 0 degreeand at 180 degrees (if the photographing angle is known accurately, theangles do not necessarily take these values), in a second turn, near 90and 270 degrees, and in a third turn, near 45, 135, 225, and 315degrees, a detailed shape is determined gradually as the number of turnsincreases.

This enables the shape estimating section to find the area for shapeestimation in an early stage of the process, which helps speed up theprocess.

Even when the photographing section photographs near 0, 10, . . . , 340,350 sequentially, making a shape estimation in the order of 0, 180, 90,270, 40, 130, 220, 310, . . . degrees produces a similar effect to thatdescribed above.

In other words, the shape estimation should be made in the followingprocedure.

For example, after shape estimation is made using an image from a firstviewpoint, shape estimation is made using an image from a secondviewpoint opposing or facing the first viewpoint.

Following this, the next shape estimation is made using an imagephotographed from a viewpoint which is one of the remaining viewpointsnot used in the shape estimation between the first viewpoint and thesecond viewpoint and which interpolates the angle formed by the visualline from the first viewpoint and the visual line from the secondviewpoint.

From this point on, a viewpoint corresponding to the position in whichthe angle formed by the visual lines from the viewpoints used in theshape estimation process is selected for the remaining images not usedin the shape estimation process sandwiched between the viewpoints of theimages used in the shape estimation. The images from the selectedviewpoint are used for the next shape estimation.

Thereafter, repeating the above processes produces the aforementionedeffect.

As shown in FIG. 34, a photographing table lift UDU that raises andlowers the object BE in the direction of rotation axis at a constantspeed may be further provided. Then, photographing may be done, whilethe turntable RU is being raised at a constant speed in response to therotation of the turntable RU.

In this case, since the relative position in the vertical directionchanges, a region of the object, which could not be estimated in asimple rotational motion, can be estimated.

Although the photographing position changes in angle and height, sincethe photographing position is moving at a constant speed, the height canbe determined by multiplying the constant speed by the photographingtime as is the angle.

Thirteenth Embodiment

The configuration of a 3-D information acquisition apparatus accordingto a thirteenth embodiment is the same as that of the twelfthembodiment.

FIG. 35 is a flowchart for the processing of the 3-D informationacquisition apparatus according to the thirteenth embodiment.

The flow of the processing of the 3-D information acquisition apparatusin the thirteenth embodiment is the same as that from step S0 to step S3in the first embodiment.

In step S4 of texture mapping, color information about the boxels oneach surface is determined using object images A01, A02, . . . , A36.

For example, the coordinates of the center of a boxel at the surface areprojected onto the object images A01, A02, . . . , A36 from which theboxel is seen. The pieces of color information about these object imagesare averaged, thereby producing color information about the boxel.

This enables not only the three-dimensional shape of the object but alsocolor information about each part to be obtained.

Fourteenth Embodiment

FIG. 36 is a block diagram showing the configuration of a 3-Dinformation acquisition apparatus according to a fourteenth embodimentof the present invention.

The fourteenth embodiment differs from the twelfth embodiment in step S2of photographing and step S3 of boundary image creation.

A light source that applies lighting so as to illuminate the entirebackground is provided behind the object BE when viewed from the cameraCU.

For example, a diffusing plate FB is provided as shown in FIG. 36 and aflash BFU is used from behind the diffusing plate in synchronizationwith photographing.

Furthermore, the object BE is placed on a transparent stand CB.

In the subsequent process, as the turntable RU is rotated as in thetwelfth embodiment, for example, 36 silhouette images S01, S02, . . . ,S36 are photographed (see FIG. 37).

Photographing in this way makes it possible to obtain the same image asthat against the sun. In the image, the background area has a highluminance and the object area is very dark.

Next, in step S3 of boundary image creation, a dark area is extractedfrom the silhouette image.

For example, the pixels with a luminance value equal to or larger than acertain threshold value are set as the background and the remaining areais set as the object.

The details of the photographing method and the cutout method are thesame as in the first to eleventh embodiments of the shape extractionsystem.

Fifteenth Embodiment

FIG. 38 is a block diagram showing the configuration of a 3-Dinformation acquisition apparatus according to a fifteenth embodiment ofthe present invention.

The fifteenth embodiment differs from the thirteenth embodiment in stepS1 of photographing, step S2 of boundary image creation, and step S4 oftexture mapping.

A light source that applies lighting so as to illuminate the entirebackground is provided behind the object BE when viewed from the cameraCU.

For example, a diffusing plate FB is provided as shown in FIG. 38 and aflash BFU is used from behind the diffusing plate in synchronizationwith photographing.

Photographing in this way makes it possible to obtain the samesilhouette image as that against the sun. In the image, the backgroundarea has a high luminance and the object area is very dark.

A flash FU for lighting the front is provided. In addition, a flashswitching unit FCU for turning on the flash FU and back flash BFUalternately or every several times is also provided.

On the turntable RU, a transparent stand CB is provided. The object BEis placed on the stand CB. As the turntable RU is rotated, textureimages T01, T02, . . . , T36 are photographed using the flash FU aplurality of times and silhouette images S01, S02, . . . , S36 arephotographed using the flash BFU a plurality of times. For example, thetexture images and silhouette images are photographed alternately (seeFIG. 39).

Next, in step S3 of boundary image creation, the dark area is extractedfrom the silhouette image.

In this case, for example, the pixels with a luminance value equal to orlarger than a certain threshold value are set as the background and theremaining area is set as the object.

The details of the photographing method and the cutout method are thesame as in the first to eleventh embodiments of the shape extractionsystem.

In the texture mapping step, not only the three-dimensional shape of theobject but also color information about each part can be obtained bygiving color information to the boxels on the basis of texture images asin the thirteenth embodiment.

Sixteenth Embodiment

A sixteenth embodiment of the present invention differs from the twelfthto fifteenth embodiments in the shape estimation step S3 in theflowchart for the 3-D information acquisition apparatus.

FIGS. 40A to 40D show two-dimensionally how boxels change in thesixteenth embodiment.

In the sixteenth embodiment, the accuracy of one boxel is set lower thana desired accuracy.

When it is determined whether a boxel is outside the object as in thetwelfth to fifteenth embodiment, a boxel whose eight vertexes are mixedinside and outside the object in determination is divided.

Although in FIGS. 40A to 40D, one square is divided into four parts forthe sake of two-dimensional explanation, the object is divided intoeight cubes in a three-dimensional space.

Then, the divided cubes are similarly subjected to outsidedetermination. In this way, the dividing process is repeatedrecursively.

When the size of the boxel becomes large enough to achieve the desiredaccuracy, the process is completed. A similar process is carried outusing the unprocessed boundary images.

With the three-dimensional reconstruction by recursive division, theobject is divided into boxels whose accuracy is lower than the desiredaccuracy except for the boundary between the inside and outside of theobject, the number of boxels to be processed decreases remarkably.

In one of the boxels once divided, when all of the boxels existing inthe boxel are determined to be outside the object, those objects arecombined into one boxel, thereby further decreasing the number ofboxels.

The determination of whether a boxel is outside the object may beapplied to not only eight vertexes but also, for example, the angularcenter of gravity of six surfaces or the midpoint of each side.

This makes it possible to reconstruct three-dimensionally an object of amore complex shape.

Seventeenth Embodiment

A seventeenth embodiment of the present invention, a modification of thetwelfth embodiment, will be explained by reference to FIG. 41.

Since the basic photographing method in the seventeenth embodiment isthe same as in the twelfth embodiment, explanation of the method will beomitted.

As shown in FIG. 41, a computer PC in which a photographing controlmodule and a 3-D shape estimating module are installed in software isconnected to the control section CB of a turntable RU, a relativemovement module, and to a camera CU, a photographing module.

The control section CB has a relative position sensing module.

The relative position sensing module, which is capable of sensing therotational position of the turntable RU, includes a sensor, such as arotary encoder or a potentiometer.

It is assumed that the computer PC is connected to the camera CU and thecontrol section CB of the turntable RU by serial communication, such asRS-232, USB, or IEEE1394, or by parallel communication, such as printerports, which enables information to be exchanged with each other.

There is no limit to these communication methods.

A subject lighting flash FU as a normal lighting module and the controlsection CB are connected to the camera CU via an X contact(sync-contact) or the like.

In addition to X-contact, some types of flash have another serialinterface for communicating with another device to inform of thecompletion of its charge.

A flash with such interface communicates with the connected flash unitat the same time and is capable of informing the information.

Hereinafter, a flash without such interface will be explained.

In this case, it is assumed that the interval between one photographingand another with the camera is long enough to charge the flash and thereis no need to check whether the flash can be used in photographing.

With the above configuration, the computer PC performs the initialsetting of the camera CU and turntable RU.

The initial setting of the camera includes, for example, exposurecondition, shutter speed, the amount of zoom, and focal length.

The initial setting of the turntable RU includes, for example, a timechart of rotational speed that takes the size, shape, weight, andmaterial of an object to be photographed into account as much aspossible to prevent the object from turning over or deforming.

After the above various initial settings have been completed and thecompletion of photographing preparation is confirmed by communication,the turntable starts to rotate. Then, the computer PC outputsphotographing request signals to the camera CU sequentially, whichcauses the camera CU to photograph a plurality of images from differentviewpoints at desired intervals of photographing.

In this case, the same signal as the X contact signal from the camera CUor the signal whose phase is the same as that of the X contact signal isinput as a photographic timing signal to the control section CB of theturntable RU.

Then, the control section CB causes the relative position sensing moduleto sense the positional information about the turntable RU at the timewhen the photographic timing signal is input (the timing with whichphotographing is done) and transmits the result to the computer PC.

Receiving the result, the computer PC records information about theposition of the viewpoint in which photographing was done and estimatesthe 3-D shape using the information about the viewpoint position and theimages picked up by the camera CU.

If the photographing request signal sent from the computer PC to thecamera is decoded by the camera quickly enough and photographing isstarted without delay, there is no problem.

Generally, however, the delay is large for the shutter timing of thecamera. Therefore, from the viewpoint of timing with which photographingis actually done, it is desirable that the X contact signal forphotographing with a flash should be used.

In the seventeenth embodiment, since the same signal as the X contactsignal or the signal whose phase is the same as the X contact signal isinputted to the control section CB, it is possible to acquireinformation about the position of the rotation of the turntable at thetime when photographing was actually done with high accuracy.

When the processing time of the camera from the time the camera receivesthe photographing instruction signal from the computer PC including thephotographing control module to the time photographing is actuallystarted is sufficiently shorter than expected or when a low accuracy canbe allowed sufficiently, the instruction signal from the photographingcontrol module may be used as the photographic timing signal withoutinputting the X contact signal from the camera to the control section CBas shown in the modification of FIG. 42.

Although the X contact signal has been used as the photographic timingsignal, a photographic timing signal using the emission of light by theflash may be used.

For example, emission of light by the flash may be received by alight-receiving module, thereby producing a photographic timing signal.

Although some types of camera do not have an X contact external flashdrive function, an internal flash could generate a photographic timingsignal by using the above function.

The camera body has the X contact, which is effective in transmitting aphotographic timing signal to the relative position sensing modulewithout cables/by radio.

(System Configuration of Final Product)

FIG. 43 is a block diagram showing the configuration of a final productof a shape extraction system including 3-D information to which thepresent invention is applied.

As shown in FIG. 43, in the configuration of a final product of a shapeextraction system including 3-D information to which the presentinvention is applied, an object to be photographed 10 is placed on atransparent photographic stand 32 supported on a turntable (orphotographic stand rotating unit) whose rotation is controlled by arotation control unit 100.

A background lighting unit 130 is provided behind the object 10.

The background lighting unit 130 is connected to a background lightinglight-source control unit 121 in a photographic condition control unit12.

The photographic condition control unit 12 includes the backgroundlighting light-source control unit 121 and an exposure control unit 122connected to a camera (or photographing unit) 11 located in front of theobject 10, a lighting unit (or subject lighting flash) 30, and arotation control unit 100.

The photographic condition control unit 12 further includes an externallight measuring unit 123, a distance measuring unit 124, a subjectphotometric unit 125, and a background luminance measuring unit 126.

The camera (photographing unit) 11 is connected to a background cutoutunit 142 via an image recording unit 141 in an image processing unit 14,such as a personal computer (PC).

The background cutout unit 142 in the personal computer (PC) isconnected to an output unit 15 via a three-dimensional reconstructingunit 143 based on a silhouette method and a texture image fusion unit144.

The background lighting unit 130 emits visible scattered light from themain surface facing the object 10 and illuminates the whole of theobject 10 from behind.

The photographic condition control unit 12 not only controls thephotographing operation of the camera 11 and its exposure condition andthe light-emitting operation and light-emitting intensity of thebackground lighting unit 130 but also supplies a background lightingtiming signal and a subject lighting timing signal to the rotationcontrol unit 100.

The photographic condition control unit 12 performs the followingcontrol: in a state where the condition control unit 12 sets specificconditions and turns on the background lighting unit 130, that is, in astate where background lighting is applied to the object 10, the controlunit 12 acquires a first image (or silhouette image) as photographicimage data obtained by photographing the object 10 with the camera(photographing unit) 11.

Furthermore, the photographic condition control unit 12 performs thefollowing control: in a state where the condition control unit 12 setsspecific conditions and turns off the background lighting unit 130, thatis, in a state where background lighting is not applied to the object10, the control unit 12 acquires a second image obtained byphotographing the object 10 with the camera 11.

In this case, the second image may be acquired by causing the lightingunit 30 to illuminate so as to photograph the second image clearly.

In the shape extraction system including 3-D information to which thepresent invention is applied, two photographic images, the first image(silhouette image) and second image, are directed to the backgroundcutout unit 142 via the image recording unit 141 in the image processingunit 14, such as a computer PC. Then, the shape of the object isextracted by the three-dimensional reconstructing unit 143 and textureimage fusion unit 144.

FIG. 44 is drawings to help explain an application using theconfiguration of FIG. 43.

That is, in the present invention, using the first image (silhouetteimage), the shape of the object is recognized and a cutout mask isformed. With this mask, the object area is cut out from the secondimage.

Specifically, as shown in FIG. 41, the first image (silhouette image) asan image with background lighting and the second image as a textureimage without background lighting are directed to the background cutoutunit 142, which creates a cutout image.

Then, this cutout image and a separately prepared desired backgroundimage are directed to the texture image fusion unit 144, which creates abackground replacement image.

The details of the photographing method and the cutout method are thesame as in first to eleventh embodiments of the shape extraction system.

The cutout image may be led to the three-dimensional reconstructing unit143, which creates a solid image reconstructed three-dimensionally by asilhouette method.

The details of the acquisition of 3-D information by thethree-dimensional reconstructing unit 143 using the silhouette methodare the same as in twelfth to seventeenth embodiments of the 3-Dinformation acquisition system.

Therefore, according to the first to eleventh embodiments of the presentinvention, it is possible provide a shape extraction apparatus andmethod which are capable of cutting out an image reliably at low costand a shape extraction system including an image cutout apparatus andmethod, and more particularly a shape extraction system using thetechnique for extracting the boundary of an object on the basis of aphotographic image obtained in a state where background lighting isapplied to the object whose two-dimensional image boundary is to beextracted and a photographic image obtained in a state where nobackground lightning is applied to the object.

Furthermore, according to the twelfth to seventeenth embodiments of thepresent invention, it is possible to provide a 3-D informationacquisition apparatus and method which are capable of not onlydetermining the boundary with high accuracy and acquiring high-accuracy3-D information but also reducing remarkably the memory capacity toacquire 3-D information about an object, shortening the photographingtime, and keeping the object stable, and a 3-D information acquisitionsystem including a 3-D information acquisition program, and moreparticularly a 3-D information acquisition system which acquires 3-Dinformation about the object by using as component techniques a shapeextraction system which extracts the two-dimensional boundary of theobject on the basis of a photographic image obtained in a state wherebackground lighting is applied to the object whose two-dimensional imageboundary is to be extracted and a photographic image obtained in a statewhere no background lightning is applied to the object.

Additional advantages and modifications will readily occur to thoseskilled in the art. Therefore, the invention in its broader aspects isnot limited to the specific details and representative embodiments shownand described herein. Accordingly, various modifications may be madewithout departing from the spirit or scope of the general inventiveconcept as defined by the appended claims and their equivalents.

1. A shape extraction apparatus comprising: a photographing module whichphotographs an object; a background lighting module which illuminatessaid object by visible light from behind with respect to saidphotographing module to identify an area including the boundary betweensaid object and the background portion in an image to be photographed bysaid photographing module; a control module which controls aphotographing operation including the exposure of said photographingmodule and the lighting intensity of said background lighting module;and an image processing module which processes the image photographed bysaid photographing module, wherein said control module sets saidexposure and lighting intensity to specific conditions so as tophotograph a processing image where the background portion on theperiphery of said object has a higher luminance than that of thesilhouette portion of said object, and said image processing moduleextracts the shape of said object by using the luminance differencebetween the silhouette area of said object and the background area onthe periphery of said object in said processing image.
 2. The shapeextraction apparatus according to claim 1, wherein said backgroundlighting module includes at least; a light-source module which emitslight in the visible light; and a light scattering module for scatteringlight from said light-source module which is provided behind said objectwith respect to said photographing module.
 3. The shape extractionapparatus according to claim 2, wherein a scattered light radiation areawhere said light scattering module radiates scattered light includes anarea corresponding to the boundary between said object and thebackground portion in the image photographed by said photographingmodule.
 4. The shape extraction apparatus according to claim 1, whereinsaid processing image is a silhouette image photographed in such amanner that said object is darker than the background.
 5. A shapeextraction method comprising: photographing an object; illuminating saidobject from behind by visible light to identify an area including theboundary between said object and the background portion in an image tobe photographed; controlling a photographing operation including theexposure in photographing said object and the lighting intensity of saidlighting; and processing said photographed image, wherein said controlsets said exposure and lighting intensity to specific conditions so asto photograph a processing image where the background portion on theperiphery of said object has a higher luminance than that of thesilhouette portion of said object, and said processing extracts theshape of said object by using the luminance difference between thesilhouette area of said object and the background area on the peripheryof said object in said processing image.
 6. An image cutout apparatuscomprising: a photographing module which photographs an object; abackground lighting module which illuminates said object by visiblelight from behind with respect to said photographing module to identifyan area including the boundary between said object and the backgroundportion in an image to be photographed by said photographing module; acontrol module which controls a photographing operation including theexposure of said photographing module and the lighting intensity of saidbackground lighting module; an image processing module which processesthe image photographed by said photographing module; and a storagemodule which stores the image processed by said image processing module,wherein said control module acquires a first image obtained by settingsaid photographing module to a first specific exposure and photographingsaid object, and a second image obtained by lighting said backgroundlighting module at a specific lighting intensity, setting saidphotographing module to said first specific exposure or a secondspecific exposure different from said first specific exposure, andphotographing said object, said image processing module extracts theshape of said object by using the luminance difference between thesilhouette portion of said object and the background portion on theperiphery of said object in said second image and creates a third imageby cutting out an image of a part corresponding to said object from saidfirst image on the basis of said extracted shape, and said storagemodule stores said third image.
 7. The image cutout apparatus accordingto claim 6, wherein said background lighting module includes at least; alight source module which emits light in the visible light; and a lightscattering module for scattering light from said light-source module,which is provided behind said object with respect to said photographingmodule.
 8. The image cutout apparatus according to claim 6, wherein ascattered light radiation area where said light scattering moduleradiates scattered light includes an area corresponding to the boundarybetween said object and the background portion in the image photographedby said photographing module.
 9. The image cutout apparatus according toclaim 6, wherein said second image is such that the background portionon the periphery of said object has a higher luminance than that of thesilhouette portion of said object.
 10. The image cutout apparatusaccording to claim 6, wherein said second image is a silhouette image ofsaid object.